Dynamic Factor Model Validation

Author

John Vivian

Table 1: Input that aaron sent me
Time Pandemic_10 Pandemic_9 Pandemic_1 Pandemic_7
0 1/1/2020 0.000000 0.000000 0.000000 0.000000
1 2/1/2020 0.000000 0.000000 0.000000 0.000000
2 3/1/2020 0.005874 0.039735 0.004223 0.060936
3 4/1/2020 0.069314 0.019868 0.002413 0.028292
4 5/1/2020 0.137923 0.026490 0.002573 0.030468

Convert input into AnnData object

AnnData object with n_obs × n_vars = 36 × 4

Manually create DataProcessor object to confirm data/inputs match expectations

Columns that fail the ADF test (non-stationary)
['Pandemic_7']
Table 2: Values have not changed as they are already normalized between 0 and 1
Pandemic_10 Pandemic_9 Pandemic_1 Pandemic_7
Time
1/1/2020 0.000000 0.000000 0.000000 0.000000
2/1/2020 0.000000 0.000000 0.000000 0.000000
3/1/2020 0.005874 0.039735 0.004223 0.060936
4/1/2020 0.069314 0.019868 0.002413 0.028292
5/1/2020 0.137923 0.026490 0.002573 0.030468

Display factor inputs to model

{'Pandemic_10': ('Pandemic',),
 'Pandemic_9': ('Pandemic',),
 'Pandemic_1': ('Pandemic',),
 'Pandemic_7': ('Pandemic',)}

Define and fit model

EM start iterations, llf=-148.46
EM iteration 10, llf=-142.49, convergence criterion=2.0537e-05
EM iteration 20, llf=-142.47, convergence criterion=6.7402e-06
EM iteration 30, llf=-142.46, convergence criterion=3.2861e-06
EM iteration 40, llf=-142.46, convergence criterion=1.9344e-06
EM iteration 50, llf=-142.46, convergence criterion=1.2716e-06
EM converged at iteration 57, llf=-142.46, convergence criterion=9.9125e-07 < tolerance=1e-06
/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/base/tsa_model.py:473: ValueWarning:

No frequency information was provided, so inferred frequency MS will be used.

Save output

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4250: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value 'Pandemic_10    0.63
Pandemic_9     0.21
Pandemic_1     0.10
Pandemic_7     0.73
Name:    idiosyncratic: AR(1), dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4250: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value 'Pandemic_10    0.54
Pandemic_9     0.29
Pandemic_1     0.47
Pandemic_7     0.11
Name: var., dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.

/home/jvivian/miniconda3/envs/py3.10/lib/python3.10/site-packages/statsmodels/tsa/statespace/dynamic_factor_mq.py:4316: FutureWarning:

Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '
Pandemic    0.77
Name:    error variance, dtype: object' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
Table 3: Raw factors.csv output
Factor_Pandemic Factor_Pandemic_10 Factor_Pandemic_9 Factor_Pandemic_1 Factor_Pandemic_7
Time
1/1/2020 1.740749 0.000000 0.000000 0.000000 0.000000
2/1/2020 1.603046 0.000000 0.000000 0.000000 0.000000
3/1/2020 1.269691 0.005874 0.039735 0.004223 0.060936
4/1/2020 1.338951 0.069314 0.019868 0.002413 0.028292
5/1/2020 1.270113 0.137923 0.026490 0.002573 0.030468
Table 4: Normalized factors + inverted
Factor_Pandemic Factor_Pandemic_10 Factor_Pandemic_9 Factor_Pandemic_1 Factor_Pandemic_7 Invert_Factor_Pandemic
0 1.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.972549 0.000000 0.000000 0.000000 0.000000 0.027451
2 0.906095 0.005874 0.039735 0.004223 0.060936 0.093905
3 0.919902 0.069314 0.019868 0.002413 0.028292 0.080098
4 0.906179 0.137923 0.026490 0.002573 0.030468 0.093821

Merge factor result back into dataframe and plot

Figure 1: Result of input (already normalized between 0-1) and pandemic factor (global=0). Inverted factor fits data well.

Summary

Wow, John is such a cool dude